Hadoop

2023-01-19 23:31| 来源: 网络整理| 查看: 265

The source of the problem: When performing the join operation on the reduce side, or when we need to avoid complexity when we aggregate data of different attribute types into a data set in multiple Mapreduce calculations, output data sets belonging to multiple value types from the mapper, is very useful. Even when we need to process different files and write multiple mappers, the data types passed in by each file are different when performing data aggregation. However, Hadoop reduce does not allow multiple input value types. In this case, the Genericwritable class can be used to wrap multiple value instances belonging to different data types.

Data example: 172.16.31.44 100 www.baidu.com 222.139.215.195 120 www.google.com 222.139.215.195 330 www.360.com 172.16.31.44 400 www.uc.net 222.139.215.org wwwpigai 50 It is separated by tabs, the first field is IP, the second field indicates the length of the request, and the third indicates the request link. In the map stage, we use IP as the key value, request word length and request link as vlaue, the purpose is to summarize the total request word length of the same IP, and output all links of the request.

Output result:

172.16.31.44 500——www.uc.net www.baidu.com 222.139.215.195 500——www.pigai.org www.360.com www.google.com

Steps:

The following steps show how to implement the Hadoop GenericWritable data type, which can encapsulate instances of data type Intwritable or Text.

Write a class extending the org.apache.hadoop.io.GenericWritable class. Implement the getType() method that returns an array of Writable classes to be used. If you want to add a custom defined constructor, then also make sure to add a no-argument default constructor. public class LogValueWritable extends GenericWritable { //Define the Writable array to be returned private static Class[] CLASS = new Class[]{ Text. class, IntWritable. class }; // Default constructor with no parameters public LogValueWritable(){} //Define the constructor of the Text type public LogValueWritable(Text text){ super.set(text); } //Define the Intwritable constructor public LogValueWritable(IntWritable intWritable){ super.set(intWritable); } //Or abandon the above input of writing each type, directly define it as Writable, so that it can automatically recognize the type /*public LogValueWritable(Writable value){ set(value); }*/ @Override protected Class

【本文地址】

公司简介

联系我们